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Abstract Body 


Background / Context 

Over the past twenty years, efforts to turn around low-performing schools have increasingly 
become a central component of federal and state education policy agendas. Spurred by federal 
legislation and funding sources such as School Improvement Grants, Race to the Top, and 
waivers from the Elementary and Secondary Education Act, the implementation of policies that 
seek to identify and turn around low-performing schools has become a major priority of every 
state education agency (SEA) in the nation. At the same time, efforts to integrate non-academic 
supports into schools have become more widespread, especially in chronically underperforming 
schools and districts. 

Despites these trends, there is limited research on the role that these efforts can play in large- 
scale school improvement, and specifically, whether programs that focus on non-academic 
supports can contribute to academic success. There is a small but growing body of literature 
demonstrating a link between programs that provide non-academic supports and student 
academic outcomes. Eor example. Child Trends conducted a review of the literature on 
integrated student support (ISS) models and reported that most rigorous quasi-experimental 
studies showed an impact of ISS approaches on student achievement (Moore, Terzian, & 
Stratford, 2014). Reviews on aspects of school climate have also shown that programs that focus 
on school safety; relationships among students, staff and families; and a culture that promotes 
strong social-emotional skills are associated with improvements in teaching and learning (Thapa, 
Cohen, Guffy, & Higgins-d’ Alessandro, 2013). 

Purpose / Objective / Research Question / Focus of Study: 

The purpose of the study was to evaluate the impact of the Wraparound Zones Initiative (WAZ), 
a program supported by the Massachusetts Department of Elementary and Secondary Education 
(ESE), on student outcomes including achievement, attendance, retention, and suspension. The 
study was conducted as part of a multi-year mixed-methods, formative and summative 
evaluation of WAZ that ESE commissioned from American Institutes for Research (AIR). 

Setting: 

The setting consisted of districts and schools in Massachusetts that had been identified by the 
state as chronically underperforming and in need of state intervention. 

Population / Participants / Subjects: 

The sample for this study was drawn from students in Cohort 1 and Cohort 2 WAZ schools 
serving elementary and/or middle grades, ^ plus students in a set of matched non-WAZ 
comparison schools. The Cohort 1 and 2 WAZ schools came from 5 of the 7 districts that were 
awarded WAZ grants. Of the two districts not included, one discontinued participation in the 
program after the first year, and the other did not begin participation until the second year and 
was therefore not obligated to participate in the evaluation. In total, the sample included 28 


^ Because only one WAZ school in the study was a traditional high school (Grades 9-12), our analysis for high 
schools outcomes would have relied on only one matched pair. We therefore removed the high school from the 
analysis and focus on elementary and middle schools only. 
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schools. Cohort 1 schools began implementation in 2011-12, and Cohort 2 schools began 
implementation in 2012-13. 

Intervention / Program / Practice: 

The WAZ Initiative is designed to create coordinated district systems that allow schools to 
proactively and systematically address students’ nonacademic needs. The four WAZ priority 
improvement areas follow: 

■ Climate and Culture. Each participating school creates a climate and a culture that 
promote mental health and positive social, emotional, and intellectual growth for 
students, resulting in a new standard of practice understood and practiced by every 
member of the school community. 

■ Identification of Student Needs and Efforts to Address Them. Each participating 
school implements a proactive system of identifying student needs in key academic and 
nonacademic areas, leading to both universal supports and targeted interventions. 

■ Community Coalitions. Each participating school integrates a range of resources to 
tailor student services from both within the school and the larger community. The range 
of services includes prevention, enrichment, early intervention, and intensive crisis 
response services 

■ District Systems of Support. Each participating district develops district-level systems 
to support the communication, collaboration, evaluation, and continuous improvement of 
the WAZ initiative. 

Six districts received WAZ grants and used them to implement the four priority improvement 
areas in a subset of schools (the number of schools per district varied from 3 to 9). Each 
participating district could implement the initiative differently based on the range of student 
needs and community assets, but the four priority improvement areas had to be met. ESE 
provided ongoing support in the form of one-on-one district technical assistance sessions, cross- 
district content meetings, district coordinator meetings, and leadership and learning exchanges. 

Research Design: 

We used a comparative interrupted time series (CITS) design to measure the impact of receiving 
a WAZ grant on student outcomes, including student achievement, attendance, retention, and 
suspension. CITS is highly regarded as one of the strongest quasi-experimental designs that can 
be used to measure program impacts in the absence of random assignment of students to a 
treatment (e.g.. Bloom, 2003; Glass, 1999; Shadish, Cook, & Campbell, 2002). Eor this 
particular study, the CITS method treats the start of the WAZ initiative as an “interruption” in 
the day-to-day operations of the school that is hypothesized to lead to an improvement in the 
identified indicators. Technically, AIR computed the deviation from the trend that occurred for 
WAZ schools upon program implementation and subtracted out any deviation from the trend that 
occurred at the same time for comparison schools. This difference in the deviation is the 
estimated treatment effect of the WAZ program. 

Data Collection and Analysis: 

We gathered extant data through ESE measuring four outcomes: (1) Student achievement, as 
measured by standardized raw scores on the English language arts (EEA) and mathematics 
sections of the Massachusetts Comprehensive Assessment System (MCAS); (2) Student 
attendance rate, calculated as number of days in attendance divided by the number of days 
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enrolled; (3) Student retention, calculated as whether the grade a student was enrolled in during the 
fall was the same grade the student was enrolled in the fall of the previous academic year; and (4) 
Suspension, calculated as whether a student received an in-school or out-of-school suspension 
during the school year. We also gathered demographic data on the students in the sample. 
Matching was conducted using the Mahalanobis D matching technique, which seeks to identify 
the optimal matched comparison school for each school based on a select set of key school-level 
indicators. 

We then employed a CITS model to evaluate the impact of WAZ on select academic (e.g., 

MCAS scores) and nonacademic (e.g., attendance, suspension rates) indicators. Specifically, AIR 
examined the change in WAZ schools’ performance when WAZ was implemented relative to the 
change for a similar set of comparison schools (selected using the matching procedures described 
earlier). In all models, the study controlled for confounding factors (e.g., student body 
characteristics) and any changes in the given indicator over time not due to the intervention 
itself. The study also accounted for the nesting of students in schools, the nesting of schools in 
matched comparison pairs, and the effect of attending a particular school nested in a particular 
matched pair in a given year (i.e., the impact of time). In addition, the study controlled for 
student- level covariates (gender, income, special education and ELL status, and race) and 
school-level factors (year of implementation, whether the school received a planning grant) and 
allowed for baseline differences between schools. 

Findings / Results: 

Overall, students in WAZ schools performed better on the Massachusetts Comprehensive 
Assessment System (MCAS) English language arts (ELA) and mathematics assessments as 
compared with students in comparison schools, when considering prior achievement trends. 
Effects were statistically significant after the second and third years of WAZ implementation for 
ELA, and after the second year for mathematics. Specifically: 

■ In the second and third years of implementation, students in WAZ schools demonstrated 
ELA scores that were 0.15 and 0.24 standard deviations higher, respectively, than what 
would be expected given prior performance trends and test score changes in non -WAZ 
comparison schools during the same time. 

■ In the second year of implementation, students in WAZ schools demonstrated 
mathematics scores that were 0.17 standard deviations higher than what would be 
expected given prior performance trends and test score changes in non-WAZ comparison 
schools during the same time. 

Although the effect for MCAS mathematics performance was not statistically significant in the 
third year, it is notable that it was of comparable magnitude to the effect size for the second year 
(0.19 standard deviations).^ 

The study also found the following: 

■ The impact of receiving a WAZ grant on academic achievement was greatest for third- 
and fourth-grade students. 


^ It is important to note that the third-year effect was only observed for schools in the first cohort that began 
implementation in 2011-12, and for which AIR had the opportunity to collect data over three years. In other 
words, Cohort 2 schools are not included in the third-year effect. The smaller sample size in the third year may 
have contributed to the lack of statistical significance for the mathematics effect. 
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■ For students with limited English proficiency, the impact of WAZ on academic 
performance was particularly strong in Year 3. 

■ There was no overall statistically significant impact of WAZ on attendance, retention, or 
suspension. 

Conclusions: 

The success of the WAZ initiative, which includes a focus on both overall school climate and 
elements of the integrated student support model (e.g., targeted supports for students, community 
partnerships), aligns well with emerging literature demonstrate a link between these wraparound- 
like supports and student outcomes. Evidence from within Massachusetts lends even further 
support to these findings. Eor example, 10 WAZ schools that began the initiative as so-called 
“Eevel 4 schools,” generally the lowest performing 2 percent schools in the state, had exited 
Eevel 4 status by the time the grant was over. In fact, among the full 2010 cohort of Eevel 4 
schools, those that were WAZ schools were more likely than non-WAZ schools to exit Eevel 4 
status by 2014 (66 percent and 40 percent, respectively). These data point to the success of WAZ 
as a component of a school turnaround strategy. 

Nonetheless, questions remain as limitations in the design prevent us from making claims about 
how and why the WAZ program led to these results. We recommend that additional research 
examine questions including the following: 

■ Which factors associated with WAZ implementation contributed the most to achievement 
gains, such as strong school climate, strong community partnerships? 

■ Will student achievement gains in WAZ schools be sustained over time when the grant 
funding ends? If so, which factors contribute to this sustainability and which act as 
barriers? 

■ In what ways does the impact of WAZ vary for students in different subgroups? 

■ What is the combined impact of WAZ with other funding streams that target low- 
performing schools (e.g.. School Redesign Grants)? 

This research study is significant in that it demonstrates that a program focused on student 
support and social-emotional learning can have an impact on student achievement, and can be an 
integral component of overall school turnaround strategy. It contributes to ongoing policy 
discussions about how to allocate limited resources towards strategies most likely to support 
rapid improvement in chronically low-performing schools. 
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